feat: add sonatype script (CM-1309) - #4299
Conversation
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
ff77df6 to
d2d4e7a
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d2d4e7a. Configure here.
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
There was a problem hiding this comment.
Pull request overview
This PR implements step 2 of 3 of the CM-1309 pipeline: a one-shot ingestion of Sonatype's Maven "popularity score" signal into the osspckgs packages table. Since Sonatype cannot deliver raw Maven download counts, they provide a normalized 0–100 popularity score (tiered P0–P3); this PR loads the first delivery (Maven P0, ~500 components) from a CSV. Because Sonatype's list reflects real-world usage, some components may not yet exist in packages, so the import inserts them (with ingestion_source='sonatype') for later backfill, while preserving ingestion_source on existing rows.
Changes:
- New one-shot script
importSonatypePopularityFromCsv.tsthat validates the CSV header, parses/validates rows (with--dry-runand--snapshot-datesupport), and applies all rows in a single transaction with insert/update/skip reporting. - New DAL function
upsertSonatypePopularity()keyed on the composite identity(ecosystem, COALESCE(namespace,''), name)that updates onlysonatype_*columns on conflict (preservingingestion_source) and detects insert-vs-update via a pre-INSERTexistingCTE. - New
IDbSonatypePopularityUpserttype describing the upsert payload.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
services/apps/packages_worker/src/maven/scripts/importSonatypePopularityFromCsv.ts |
New CSV import script: header validation, per-row parse/validate, single-transaction upsert, audit reporting of newly inserted packages. |
services/libs/data-access-layer/src/osspckgs/packages.ts |
Adds upsertSonatypePopularity() — composite-key upsert that preserves ingestion_source and returns insert-vs-update. |
services/libs/data-access-layer/src/osspckgs/types.ts |
Adds IDbSonatypePopularityUpsert type for the new upsert payload. |
Notes verified during review: the composite conflict target matches the existing unique index (initial_schema.sql:82); all NOT NULL packages columns omitted from the new INSERT have defaults (created_at gained DEFAULT NOW() in V1780600000), so inserts won't fail; and the pre-INSERT existing CTE reliably detects insert-vs-update since data-modifying CTEs share one snapshot.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org> Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>
Summary
Adds the one-shot ingestion of the Sonatype popularity signal into the osspckgs
packagestable.Sonatype could not deliver raw monthly Maven download counts, so instead they provide a popularity score derived from their SCA telemetry — normalized 0–100 per ecosystem, tiered P0–P3. First delivery is Maven P0 only (top ~500 components). This PR loads that CSV into the
sonatype_*columns.This is step 2 of 3:
sonatype_*columns onpackages.sonatype_popularity_scoreintorank_packages()as a 4th signal.Because Sonatype's list is "what industry actually uses", some rows don't exist in
packagesyet (deps.dev hasn't backfilled them). The script inserts them if missing so the signal isn't dropped — the row is then promoted to critical by the ranking pass and backfilled by the daily Maven enrichment worker.Changes
services/apps/packages_worker/src/maven/scripts/importSonatypePopularityFromCsv.ts(follows the existingimportMaintainersFromCsv.tspattern, reusesparseCsv):<input.csv>,--snapshot-date=YYYY-MM-DD(derived from the filename if omitted),--dry-run.component_keyagainstnamespace:name.upsertSonatypePopularity()indata-access-layer/src/osspckgs/packages.ts— a dedicated upsert (notupsertPackage) because:ingestion_sourceon conflict (existing rows keep how they were ingested);upsertPackageoverwrites it;sonatype_*columns on insert (ingestion_source='sonatype'), leaving the rest for later backfill.(ecosystem, COALESCE(namespace,''), name), notpurl— so updating the ~471 existing rows never depends on the stored purl format. New rows get a version-stripped purlpkg:maven/{groupId}/{artifactId}, matching how Maven purls are stored (verified against the DB).existingCTE (robust; avoids thexmax=0internals hack).IDbSonatypePopularityUpsertindata-access-layer/src/osspckgs/types.ts.Type of change
JIRA ticket
CM-1309
Note
Medium Risk
Writes directly to the core
packagestable and can insert new stub rows, but scope is a small idempotent one-shot import with validation, dry-run, and transactional all-or-nothing semantics.Overview
Adds one-shot ingestion of Sonatype’s Maven popularity CSV (score, rank, tier) into osspckgs
packagessonatype_*columns, as step 2 before wiring the signal intorank_packages().A new
importSonatypePopularityFromCsvscript (same family as maintainer CSV import) parses and validates the Sonatype header/rows, supports--dry-runand--snapshot-date, runs the load in a single transaction, and logs inserted vs updated rows plus any new package stubs created from the feed.The DAL adds
upsertSonatypePopularityandIDbSonatypePopularityUpsert: upserts on(ecosystem, namespace, name)(notpurl), updates onlysonatype_*on conflict (ingestion_sourceunchanged), and inserts minimal rows (ingestion_source='sonatype', version-stripped Mavenpurl) when a component isn’t inpackagesyet.package.jsongainsimport:sonatype-popularitynpm scripts to run it locally or in prod.Reviewed by Cursor Bugbot for commit bee6a29. Bugbot is set up for automated code reviews on this repo. Configure here.